Surgery offers the potential to treat and cure many diseases, but complications from surgical procedures remain the third highest cause of death globally [1]. Recent studies have shown that surgeons rated as higher-skilled via peer grading have lower rates of complications and death [2, 3]. Systems to evaluate surgical skill and provide feedback to improve technique could have a dramatic effect on the variation that exists in the field. Unfortunately, current approaches for evaluating surgical procedures and technique are primarily qualitative and do not have the ability to scale or even identify the elements of surgeon judgment that drive patient outcomes [4]. Artificial intelligence (AI) in the form of computer vision algorithms could provide scalable, automated analysis of surgical behaviors from video streams. AI could serve as an additional coach for surgical trainees and as an expert colleague for experienced surgeons [5]. However, the development of computer vision for open surgery--the dominant form of surgery defined as traditional, non-camera-based surgical techniques [6]--has been limited by two factors: the complexity of the AI task, and a lack of diverse and sizable training datasets [7]. Our work shows that a multi-task, spatiotemporal AI model, trained on multi-institutional data from numerous surgeons, has the potential to provide consistent analysis and feedback without the bias of any particular surgeon's experience.